Introduction to Machine Learning

Unit 22: Neural Networks - Advanced Topics

Introduction

Welcome to Unit 22, where we continue our exploration of neural networks and dive into advanced topics that make them powerful for real-world applications.

Today's Focus:

  • Model Comparison: Linear vs Nonlinear Classification
  • Activation Functions: Linear, Sigmoid, Tanh, ReLU, Softmax
  • Regularization: Dropout, L1/L2, Early Stopping
  • Architecture Design: Choosing layers, units, and hyperparameters
  • Optimization: Learning rate strategies, SGD variants

This lecture builds upon Unit 21's introduction to the backward pass and explores how to design effective neural networks and train them efficiently.

Theory

Model Comparison: Linear vs Nonlinear Classification

Let's compare different models on classification tasks to understand their strengths and limitations.

# Define models for comparison models = { 'Logistic Regression': LogisticRegression(), 'Decision Tree': DecisionTreeClassifier(max_depth=5, random_state=42), 'GradientBoosting': GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42), 'Neural Network': MLPClassifier(hidden_layer_sizes=(10,), max_iter=1000, random_state=42) }
Model Comparison Visualization Comparison of logistic regression, decision tree, gradient boosting, and neural network decision boundaries on the same classified data. Model Comparison Visualization How different models classify the same data through increasingly expressive decision boundaries Class 0 Class 1 Logistic Regression (Linear boundary) Class 0 Class 1 Straight line separating classes Accuracy ~0.850 Decision Tree (Piecewise linear) Class 0 Class 1 Step function separating classes Accuracy ~0.920 Gradient Boosting (Complex boundary) Class 0 Class 1 Smooth curve separating classes Accuracy ~0.940 Neural Network (Nonlinear boundary) Class 0 Class 1 Complex curve separating classes Accuracy ~0.950 (with 1 hidden layer) KEY INSIGHT Neural networks can learn complex, nonlinear decision boundaries that simpler model architectures cannot capture.

Observations:

  • Logistic Regression: Creates linear decision boundaries. Limited to linearly separable problems.
  • Decision Tree: Creates piecewise linear boundaries. Can handle some nonlinearity but may overfit.
  • Gradient Boosting: Creates complex, smooth boundaries. Very powerful but can be slow.
  • Neural Network: Can learn highly complex, nonlinear boundaries. Most flexible but requires careful tuning.

Important: Neural network configuration needs enhancements. A single hidden layer with 10 units may not be sufficient for complex problems. We'll explore how to improve this.

Activation Functions - Overview

Different activation functions serve different purposes in neural networks. The choice depends on:

Linear Activation

\[ f(z) = z \]

Properties:

  • No transformation: Identity function
  • Range: \((-\infty, +\infty)\)

Usage:

  • Output layer in regression problems where we need unbounded predictions
  • Rarely used in hidden layers (would collapse the network to linear regression)

Sigmoid (Logistic) Activation

\[ \sigma(z) = \frac{1}{1 + e^{-z}} \]

Properties:

  • Range: (0, 1)
  • Output interpretable: As probability
  • Drawback: Derivative saturates (becomes very small) for large \(|z|\), causing vanishing gradients

Derivative:

\[ \sigma'(z) = \sigma(z) \cdot (1 - \sigma(z)) = a \cdot (1 - a) \]

Usage:

  • Output layer in binary classification (probability of positive class)
  • Sometimes in hidden layers, but not recommended for deep networks due to vanishing gradient problem

Vanishing Gradient Problem

During backpropagation, gradients get multiplied as they flow backward through layers. If these gradients are very small (\(< 1\)), they get smaller and smaller with each layer, eventually becoming nearly zero.

Why it happens: Early layers (close to input) barely learn anything because their gradients are too tiny to cause meaningful weight updates.

Chain rule in backpropagation:

\[ \frac{\partial L}{\partial w^{[1]}} = \frac{\partial L}{\partial z^{[3]}} \cdot \frac{\partial z^{[3]}}{\partial a^{[2]}} \cdot \frac{\partial a^{[2]}}{\partial z^{[2]}} \cdot \frac{\partial z^{[2]}}{\partial a^{[1]}} \cdot \frac{\partial a^{[1]}}{\partial z^{[1]}} \cdot \frac{\partial z^{[1]}}{\partial w^{[1]}} \]

Problem with sigmoid:

  • Maximum value of \(\sigma'(z)\) is 0.25 (when \(z = 0\))
  • For \(|z| > 3\), \(\sigma'(z) < 0.05\) (very small!)

Example in a 5-layer network:

  • Layer 5 gradient: 0.2
  • Layer 4: \(0.2 \times 0.2 = 0.04\)
  • Layer 3: \(0.04 \times 0.2 = 0.008\)
  • Layer 2: \(0.008 \times 0.2 = 0.0016\)
  • Layer 1: \(0.0016 \times 0.2 = 0.00032\) ← Almost zero!

Result: Early layers learn very slowly or not at all.

Tanh (Hyperbolic Tangent) Activation

\[ \tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}} \]

Properties:

  • Range: (-1, 1)
  • Zero-centered: Unlike sigmoid (which is always positive)
  • Stronger gradients: Than sigmoid near zero
  • Drawback: Still suffers from vanishing gradients for large \(|z|\)

Derivative:

\[ \tanh'(z) = 1 - \tanh^2(z) \]

Usage:

  • Hidden layers (better than sigmoid due to zero-centering)
  • Not recommended for very deep networks

ReLU (Rectified Linear Unit) Activation

\[ \operatorname{ReLU}(z) = \max(0, z) \]

Properties:

  • Range: [0, ∞)
  • Computationally efficient: Simple threshold operation
  • Does not saturate: For positive values (derivative = 1)
  • Drawback: "Dying ReLU" problem when \(z < 0\) (gradient = 0)

Derivative:

\[ \operatorname{ReLU}'(z) = \begin{cases} 1 & z > 0 \\ 0 & z \leq 0 \end{cases} \]

Usage:

  • Default choice for hidden layers in deep feedforward networks
  • Most popular activation function in modern deep learning

Dying ReLU Problem: If a ReLU neuron's output is always negative (z ≤ 0), its gradient will always be zero, and the neuron will never update its weights. This neuron is effectively "dead."

Solutions:

  • Use a small positive bias in initialization
  • Use Leaky ReLU: \(f(z) = \max(\alpha z, z)\) where \(\alpha\) is small (e.g., 0.01)
  • Use Parametric ReLU (PReLU): Learn \(\alpha\) during training

Softmax Activation

\[ \operatorname{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}} \]

Properties:

  • Converts vector: Of K real numbers into probability distribution
  • All outputs sum to 1: \(\sum_{i=1}^K \operatorname{softmax}(z_i) = 1\)
  • Range: (0, 1) for each output

Usage:

  • Output layer only in multiclass classification (K > 2 classes)
  • Produces probability for each class

Why Softmax Uses Exponentials

The exponential function amplifies differences and makes the model more confident in its predictions.

Example: Given output vector: [1, 2, 3]

  • Simple normalization: [1/6, 2/6, 3/6] = [0.167, 0.333, 0.500]
    • Differences are preserved linearly
  • Softmax (with exponentials): \[ \text{softmax}([1,2,3]) = \left[\frac{e^1}{e^1 + e^2 + e^3}, \frac{e^2}{e^1 + e^2 + e^3}, \frac{e^3}{e^1 + e^2 + e^3}\right] \approx [0.09, 0.24, 0.67] \]
    • The largest value (3) gets amplified to dominate the distribution

Activation Function Summary

Layer Type Problem Type Recommended Activation
Hidden layers Any ReLU (default)
Tanh (alternative)
Sigmoid (not recommended for deep networks)
Output layer Regression Linear (no activation)
Binary classification Sigmoid
Multiclass classification Softmax

Key Takeaways for Activation Functions:

  • Use ReLU in hidden layers for most cases
  • Choose output activation based on your problem type
  • Avoid sigmoid/tanh in deep networks with many layers (vanishing gradients)
  • ReLU is preferred in deep networks due to its non-saturating property for positive inputs

Regularization Methods

Regularization methods are techniques used to prevent overfitting and improve the generalization of neural networks.

How they work: Introduce constraints or penalties on model parameters so that the model does not become unnecessarily complex and fits noise in the training data.

In neural networks, the three most widely used regularization techniques are:

  1. Dropout
  2. L1 / L2 Regularization
  3. Early Stopping

Dropout

Dropout randomly "ignores" a subset of hidden units during training.

How Dropout Works:

  • At each training iteration, each hidden node is independently assigned a Bernoulli random variable:
    • 1 → keep the node
    • 0 → drop the node
  • Dropped nodes do not participate in the forward pass (their outputs are zeroed)
  • In the backward pass, their weights are not updated
  • Thus, every training iteration effectively uses a different, randomly-thinned network

Dropout Rate:

  • The fraction of units dropped at each iteration
  • Typical values: 0.1-0.5 (rarely more)
  • Example (Keras): model.add(Dropout(0.25))

Important: Dropout is only active during training, not during inference (prediction). At test time, all units are used, but their outputs are scaled by the dropout rate to maintain expected output magnitudes.

Dropout Visualization Comparison of an original neural network with two different subnetworks created by dropout at a rate of 0.5. Dropout Visualization Randomly removing hidden units creates a different sub-network at each iteration. Original Network FULL MODEL h1 h2 h3 Output With Dropout RATE = 0.5 h1 h2 h3 Output dropped Another Iteration NEW MASK h1 h2 h3 Output dropped i KEY INSIGHT Each iteration uses a different sub-network, preventing co-adaptation of features and improving generalization.

L1/L2 Regularization

We already covered ridge, lasso, and elastic net in regression. The same mathematical idea carries into neural networks:

L2 Regularization (Ridge):

  • Effect: Encourages small, diffuse weights → smoother functions
  • The sum runs over all weights in all layers. Biases are usually excluded.
  • If the original loss is \(L(y, \hat{y})\), then with L2 regularization:
\[ \mathcal{J}(x, y, w, b) = \mathcal{L}(y, \hat{y}) + \lambda \sum_{i} w_{i}^{2} \]

L1 Regularization (Lasso):

  • Effect: Encourages sparse weights → some weights become exactly zero
\[ \mathcal{J}(x, y, w, b) = \mathcal{L}(y, \hat{y}) + \lambda \sum_{i} |w_{i}| \]

Note: In neural networks, L2 is used far more commonly than L1.

Early Stopping

Early stopping stops training before the model begins to overfit.

How Early Stopping Works:

  1. Split data into training and validation sets
  2. During training, monitor the validation loss
  3. If validation loss stops improving (e.g., for 5 epochs), training is terminated

Interpretation:

  • Early stopping is effectively a regularizer on the number of training steps
  • Models trained too long tend to overfit; stopping earlier keeps the model in a "simpler" region of parameter space
Early Stopping Visualization A chart showing training loss decreasing over epochs while validation loss decreases and then increases, indicating the optimal point to stop training before overfitting. Early Stopping Visualization Training loss vs. validation loss over epochs Loss Epochs 0 10 20 30 40 50 60 Training loss Validation loss Underfitting Both losses are improving Optimal stopping point Validation loss is at its minimum Overfitting begins Validation loss starts to rise ✓ Key insight Stop training when validation loss starts to increase. This signals that the model is beginning to overfit the training data.

Classification Example

Consider a classification problem with the following features:

Obs. ALCHL_I PROFIL_I_R SUR_COND VEH_INVL MAX_SEV_IR
111111
221110
321111
411110
521112
620111
720131
820141
920120
1020120
Feature Description
ALCHL_IPresence (1) or absence (2) of alcohol
PROFIL_I_RProfile of the roadway: level (1), other (0)
SUR_CONDSurface condition of the road: dry (1), wet (2), snow/slush (3), ice (4), unknown (9)
VEH_INVLNumber of vehicles involved
MAX_SEV_IRPresence of injuries/fatalities: no injuries (0), injury (1), fatality (2)

To use a neural net architecture for this classification problem:

Guidelines for Choosing Architecture

For tabular data, 1-2 hidden layers are typically sufficient:

  • Universal Approximation Theorem: A single hidden layer can capture complex non-linear relationships between predictors
  • Size of hidden layers: The number of nodes determines the network's capacity:
    • Too few nodes: → underfitting (can't capture complexity)
    • Too many nodes: → overfitting (memorizes training data)

Rule of thumb for tabular data:

  • Start with p to 2p nodes (where p = number of input features)
  • Or try common sizes: 32, 64, 128 nodes
  • Monitor validation performance and adjust
  • Use regularization techniques (dropout, early stopping) to control overfitting

Choosing an Architecture (Cont'd)

Number of output nodes:

  • For classification (categorical outcome with m classes):
    • Use m nodes with softmax activation (most common)
    • Or m-1 nodes (the m-th class probability is implicit)
  • Special case - Binary classification:
    • Often use 1 node with sigmoid activation
  • For regression (numerical outcome):
    • Use 1 node with linear activation (no activation function)
    • Use k nodes if predicting k different numerical targets simultaneously (multi-output regression)

Beyond Tabular Data

While 1-2 hidden layers work well for tabular data, other data types require deeper architectures:

  • Image data (Computer Vision):
    • Architecture: CNNs with 50-200+ layers (e.g., ResNet, VGG)
    • Features learned: Hierarchical visual features: edges → textures → parts → objects
  • Text data (Natural Language Processing):
    • Architecture: Transformers with 12-96+ layers (e.g., BERT, GPT)
    • Features learned: Complex linguistic patterns and long-range dependencies

Learning Rate

The learning rate controls how much we adjust weights during each update. Choosing the right strategy is crucial for successful training.

Strategy 1: Fixed Learning Rate

  • Description: Keep the learning rate constant throughout training (e.g., \(\eta = 0.001\))
  • Advantage: Simple, no tuning needed
  • Disadvantage: May be too large (oscillate around minimum) or too small (slow convergence)

Strategy 2: Learning Rate Decay/Scheduling

Description: Start with larger value (\(\eta_0\)), gradually decrease over time

Rationale: Learn quickly initially, then fine-tune as weights become more reliable

Common schedules:

  • Step decay: Reduce by factor (e.g., ÷5) every N iterations
  • Exponential decay: \(\eta = \eta_0 \cdot e^{-kt}\)
  • 1/t decay: \(\eta = \eta_0 / (1 + kt)\) where \(t =\) iteration number

Strategy 3: Adaptive/Performance-Based

  • Description: Monitor the loss function during training
  • Rule: As long as loss is decreasing, keep current learning rate
  • When loss plateaus (stops decreasing for a set number of iterations), reduce learning rate (e.g., divide by 5 - sklearn default)
  • This allows network to escape plateaus and find better solutions

Strategy 4: Adaptive Optimizers (Modern Default)

  • Description: Use optimizers that automatically adjust learning rates per parameter
  • Examples: Adam, RMSprop, AdaGrad
  • Mechanism: Maintain different learning rates for each weight, adapt based on gradient history
  • Usage: Most common choice in modern deep learning

Weight Initialization

Initializing the weights and biases intelligently is crucial for ensuring the model's convergence during training. Poor initialization can lead to issues such as slow convergence, getting stuck in local minima, or vanishing/exploding gradients.

  • Zero Initialization: Setting all weights to zero is a common but not always the best strategy. All neurons in a layer will compute the same output and update identically, preventing the network from learning asymmetric features.
  • Random Initialization: Initialize weights with small random values. The random values are usually drawn from a normal distribution (Gaussian) or a uniform distribution.

Xavier/Glorot Initialization:

  • Sets the weights using a normal distribution with a mean of 0 and a variance of \(2 / (\text{number of input and output units})\)
  • Effective for sigmoid and hyperbolic tangent (tanh) activation functions

He Initialization:

  • Similar to Xavier, but with a variance of \(2 / \text{number of input units}\)
  • Often used with rectified linear unit (ReLU) activation functions

Batch, Mini-Batch, and SGD

Different approaches to gradient descent affect training efficiency and convergence:

  • Batch (Full-Batch) Gradient Descent:
    • Computes gradients over the entire training set before updating weights and biases
    • Pros: Stable convergence, exact gradient
    • Cons: Computationally expensive for large datasets, requires loading all data into memory
  • Stochastic Gradient Descent (SGD):
    • Update parameters after each individual training example
    • Pros: The "noisy" updates can help escape local minima
    • Cons: May lead to slower convergence due to high variance in the gradient estimates
  • Mini-Batch SGD:
    • Use a subset (mini-batch) of the training data to compute the gradient and update
    • The mini-batch size is a hyperparameter (e.g., 32, 64, 128)
    • Pros: Balances the stability of batch gradient descent and the efficiency of SGD
    • Cons: Still has some noise in gradient estimates

Training dynamics:

  • On each epoch (a full pass through data), parameters may be updated many times if using SGD or mini-batch
  • For SGD: Number of updates per epoch = number of training examples
  • For mini-batch SGD: Number of updates per epoch = number of batches
Gradient Descent Variants Comparison of Batch, Stochastic, and Mini-Batch Gradient Descent, including their process, advantages, disadvantages, and typical batch sizes. Gradient Descent Variants Three approaches to computing gradients and updating model weights Batch Gradient Descent Full dataset per optimization step 1 Entire dataset Compute gradients over ALL samples 2 Update weights ONCE per epoch PROS Stable, exact gradient CONS Slow, memory intensive Stochastic Gradient Descent One sample per optimization step 1 Single sample Compute gradients over ONE sample 2 Update weights FOR EACH sample PROS Fast, can escape local minima CONS Noisy, may not converge Mini-Batch Gradient Descent A small group per optimization step 1 Mini-batch (e.g., 32–128 samples) Compute gradients over batch 2 Update weights PER batch PROS Balance of speed and stability CONS Still some noise Typical batch sizes Choose based on dataset scale and available memory Small datasets: 16, 32 Medium datasets: 32, 64, 128 Large datasets: 128, 256, 512

Momentum

Standard gradient descent can be slow in valleys (long, narrow regions) and oscillate in steep directions.

Standard Gradient Descent:

\[ \theta_{\text{new}} = \theta_{\text{old}} - \eta \cdot \nabla L(\theta) \]

Momentum:

Adds "inertia" to updates by accumulating past gradients, like a ball rolling downhill.

\[ v_{\text{new}} = \beta \cdot v_{\text{old}} + \nabla L(\theta) \] \[ \theta_{\text{new}} = \theta_{\text{old}} - \eta \cdot v_{\text{new}} \]

Benefits:

  • Speeds up convergence in consistent gradient directions
  • Reduces oscillations and helps escape shallow local minima

Note: In practice, modern optimizers like Adam incorporate momentum-like mechanisms automatically, so you rarely need to implement it manually.

Try It Yourself

Problem 1: Activation Function Selection

You are building a neural network for each of the following tasks:

  1. Predicting house prices (regression)
  2. Binary classification (spam detection)
  3. Multiclass classification (handwritten digit recognition)

Task: What activation function would you use for the output layer in each case?

Solution:

  1. House price prediction (regression): Linear (no activation function)
  2. Spam detection (binary classification): Sigmoid
  3. Digit recognition (multiclass classification): Softmax
Problem 2: ReLU Derivative

Given the ReLU activation function \(f(z) = \max(0, z)\), calculate the derivative for the following inputs:

  1. z = 2
  2. z = -1
  3. z = 0

Solution:

Using the derivative definition:

\[ \operatorname{ReLU}'(z) = \begin{cases} 1 & z > 0 \\ 0 & z \leq 0 \end{cases} \]
  1. z = 2: Since 2 > 0, ReLU'(2) = 1
  2. z = -1: Since -1 ≤ 0, ReLU'(-1) = 0
  3. z = 0: Since 0 ≤ 0, ReLU'(0) = 0
Problem 3: Softmax Calculation

Calculate the softmax for the following input vector:

z = [1, 2, 3]

Task: Compute softmax(z)

Solution:

Using the softmax formula:

\[ \operatorname{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}} \]

Step 1: Compute exponentials:

  • e^1 ≈ 2.718
  • e^2 ≈ 7.389
  • e^3 ≈ 20.086
  • Sum = 2.718 + 7.389 + 20.086 ≈ 30.193

Step 2: Compute softmax for each element:

  • softmax(1) = 2.718 / 30.193 ≈ 0.090
  • softmax(2) = 7.389 / 30.193 ≈ 0.245
  • softmax(3) = 20.086 / 30.193 ≈ 0.665

Verification: 0.090 + 0.245 + 0.665 ≈ 1.000 ✓

Problem 4: Dropout Implementation

You have a hidden layer with 100 neurons and want to apply dropout with a rate of 0.25.

Tasks:

  1. How many neurons will be kept (on average) in each training iteration?
  2. What is the probability that a specific neuron is dropped?
  3. During inference (testing), if a neuron has an activation of 0.8, what will be its scaled output?

Solution:

  1. Neurons kept: 100 × (1 - 0.25) = 75 neurons (on average)
  2. Probability of dropping: 0.25 (dropout rate)
  3. Scaled output during inference: At test time, dropout is turned off, but outputs are scaled by the dropout rate to maintain expected values. So 0.8 × (1 - 0.25) = 0.8 × 0.75 = 0.6
Problem 5: Learning Rate Scheduling

You are training a neural network with an initial learning rate of \(\eta_0 = 0.1\).

Tasks:

  1. Using step decay with a factor of 0.5 every 100 iterations, what is the learning rate at iteration 250?
  2. Using exponential decay with \(k = 0.01\), what is the learning rate at iteration 100?
  3. Using 1/t decay with \(k = 0.1\), what is the learning rate at iteration 50?

Solution:

  1. Step decay: At iteration 250, we've passed 2 decay points (100 and 200). Learning rate = 0.1 × (0.5)^2 = 0.1 × 0.25 = 0.025
  2. Exponential decay: \(\eta = \eta_0 \cdot e^{-kt} = 0.1 \cdot e^{-0.01 \times 100} = 0.1 \cdot e^{-1} \approx 0.1 \times 0.3679 = \) 0.03679
  3. 1/t decay: \(\eta = \eta_0 / (1 + kt) = 0.1 / (1 + 0.1 \times 50) = 0.1 / (1 + 5) = 0.1 / 6 \approx \) 0.01667

Interactive Quiz

Test your understanding of Neural Networks Advanced Topics:

Question 1: Which activation function is most commonly used in hidden layers of deep neural networks?

A) Sigmoid
B) ReLU
C) Tanh
D) Linear

Question 2: What is the primary problem with sigmoid activation in deep networks?

A) It's computationally expensive
B) Vanishing gradients
C) It only outputs positive values
D) It's not differentiable

Question 3: Which regularization technique randomly drops neurons during training?

A) L1 Regularization
B) L2 Regularization
C) Dropout
D) Early Stopping

Question 4: Which activation function should be used for the output layer in a multiclass classification problem?

A) Sigmoid
B) Linear
C) Softmax
D) ReLU

Question 5: What is the main advantage of mini-batch SGD over batch SGD?

A) More accurate gradient estimates
B) Faster training and lower memory usage
C) Guaranteed convergence to global minimum
D) No need for hyperparameter tuning

Key Takeaways

Activation Functions:

  • Linear: f(z) = z, range: (-∞, ∞), used for regression output layers
  • Sigmoid: σ(z) = 1/(1+e^-z), range: (0,1), used for binary classification output layers, avoids in deep hidden layers
  • Tanh: range: (-1,1), zero-centered, better than sigmoid for hidden layers but still has vanishing gradients
  • ReLU: max(0,z), range: [0,∞), most popular for hidden layers, computationally efficient, non-saturating for positive values
  • Softmax: Converts vector to probability distribution, used for multiclass classification output layers

Vanishing Gradient Problem:

  • Gradients become extremely small in early layers of deep networks
  • Caused by repeated multiplication of small gradients through chain rule
  • Sigmoid and tanh are particularly susceptible (derivatives saturate)
  • ReLU helps mitigate this problem for positive inputs

Regularization:

  • Dropout: Randomly drops neurons during training, prevents co-adaptation, typical rate: 0.1-0.5
  • L1 Regularization: Encourages sparse weights, some weights become exactly zero
  • L2 Regularization: Encourages small, diffuse weights, more common in neural networks
  • Early Stopping: Stops training when validation loss stops improving, prevents overfitting

Architecture Design:

  • For tabular data: 1-2 hidden layers are typically sufficient
  • Hidden layer size: Start with p-2p nodes (p = input features) or try 32, 64, 128
  • Output layer: Softmax for multiclass, sigmoid for binary, linear for regression
  • For other data types: CNNs for images, Transformers for text

Optimization:

  • Learning rate strategies: Fixed, decay, adaptive, or adaptive optimizers (Adam)
  • Weight initialization: Xavier/Glorot for sigmoid/tanh, He for ReLU
  • Gradient descent variants: Batch (stable but slow), SGD (noisy but fast), Mini-batch (balanced)
  • Momentum: Adds inertia to updates, speeds up convergence, reduces oscillations

Common Pitfalls

⚠️ Activation Functions:

  • Using sigmoid in deep networks: Can cause vanishing gradients, early layers learn very slowly
  • Using ReLU without care: Can cause "dying ReLU" problem if many neurons have negative inputs
  • Using softmax in hidden layers: Softmax should only be used in the output layer for multiclass classification
  • Using linear activation in hidden layers: Collapses the network to a linear model, losing the benefits of deep learning
  • Not matching output activation to problem: Using sigmoid for regression or linear for classification

⚠️ Regularization:

  • Using dropout in output layer: Dropout should typically only be applied to hidden layers
  • Dropout rate too high: Can cause underfitting, typical range is 0.1-0.5
  • Dropout during inference: Dropout should be turned off during testing/prediction
  • Early stopping too early: May stop before the model has learned useful patterns
  • Early stopping too late: May allow the model to overfit
  • L1/L2 regularization strength: λ too large can cause underfitting, λ too small may not prevent overfitting

⚠️ Architecture Design:

  • Too few hidden units: May not have enough capacity to learn complex patterns (underfitting)
  • Too many hidden units: May overfit the training data, slow to train
  • Too many layers for simple problems: Unnecessary complexity, may overfit
  • Not using regularization: Deep networks with many parameters are prone to overfitting
  • Fixed architecture: Not experimenting with different architectures to find the best one

⚠️ Optimization:

  • Learning rate too large: Can cause weights to oscillate or diverge
  • Learning rate too small: Can lead to very slow convergence
  • Poor weight initialization: Can lead to slow convergence or getting stuck in poor local minima
  • Batch size too small: Can lead to noisy gradient estimates and slow convergence
  • Batch size too large: Can be memory-intensive and slow
  • Not using momentum: Can lead to slow convergence in valleys and oscillations in steep directions

Resources

📚 Neural Networks:

📚 Activation Functions:

📚 Regularization:

📚 Optimization:

📖 Books:

💻 Practical Implementation: